1 Objetivo

O objetivo desse notebook é efetuar todo o processo de modelagem da base de dados adult, disponibilizada para o desafio do curso de introdução ao Machine Learning da Curso-R, utilizando o framework tidymodels. Ou seja, explorar, tratar, preparar, tunnar e escolher o modelo que melhor se ajusta aos dados disponibilizados. Vamos nessa!

2 Leitura da base

2.1 Informações preliminares


As variáveis parecem estar com formatos corretos. Ponto de atenção para as variáveis wokclass, occupation e native_country, que apresentam valores missing.

3 AED


Agora vamos analisar o comportamento das variáveis para definirmos como tratar os nossos dados para o modelo.

3.1 Análise bivariada



# DataExplorer::create_report(adult)

devtools::source_url("https://raw.githubusercontent.com/ricardomattos05/functions/master/function_AED_bivariada.R")
# 
# 
adult2 <- adult %>%
            select(-id) %>% 
            mutate(resposta = if_else(resposta == ">50K", 1, 0))
# 
# 

# names(adult2)
for (i in 1:(length(adult2)-1) ) {
  
  df <- adult2[,c(i,15)]
  cat("### ",names(df[,1]),"\n") 
  print(AED_biv(df,glue("resposta"),"Pre"))
  cat('\n\n')
}

Observações:

  • education : é possível visualizar que quanto maior o grau de escolaridade, maior a proporção de pessoas com salarios acima de 50k. E que as categorias abaixo de HS-grad, 1th-4th até 12thalém de serem pouco representativas, possuem baixa proporção, vamos então criar uma categoria uma nova consolidando elas HS-not-grad.

  • marital_status : aqui iremos agrupar os campos Married-AF-spouse e Married-civ-spouse, criando a categoria Married, baseado na similaridade entre elas com relação a variável resposta e considerando a descrição delas.

  • native_country : É um campo com pouca variabilidade, onde 90% dos dados estão atribuídos como “Estados Unidos”. Sendo assim, poderia considerar apenas Estados Unidos e agrupar o restante como outros, mas vamos manter o máximo de informação e reduzir as categorias para 3, agrupando todos os países que obtiveram proporção maior que a média, manter o valor mais representativo e uma categoria com os países abaixo da média.

  • relationship : campo contém os campos husband e wife, aparentemente poderiamos agrupa-los, vamos analisar mais afundo.

  • capital_loss e capital_gain : Aparentemente tanto quem ganha quanto quem perde algum valor apresentam maiores probabilidades de ter salario >50k. Vamos então avaliar a correlação entre elas.

  • workclass : Categorias com baixa representatividade como Never-workede Without-pay não possuem classificação com a resposta de interesse “>50k”, vamos dar um zoom nessa variável e analisar os NA’s que identificamos também.

3.2 AED final:

3.2.1 occupation


ggplot(adult, aes(x = occupation, fill = resposta)) + 
  geom_bar(position="fill") + 
  theme(axis.text.x = element_text(angle = 90)) + 
  ggtitle("occupation")


É possível ver que não faria sentido atribuir os NAs de forma modal, uma vez que nosso objetivo é obter o maior poder preditivo possível, logo, não queremos perder informação. Sendo assim, não vamos diluir os NAs na categoria com maior representatividade Prof-specialty, vamos atribuir à uma categoria com proporções similares e que possui uma boa representatividade, Farming-fishing.

3.2.2 relationship


ggplot(adult, aes(x = relationship)) +
  geom_bar() +
  theme(axis.text.x = element_text(angle = 90)) + 
  ggtitle("relationship")

ggplot(adult, aes(x = relationship, fill = resposta)) + 
  geom_bar(position="fill") + 
  theme(axis.text.x = element_text(angle = 90)) + 
  ggtitle("relationship")

ggplot(adult, aes(x = relationship, fill = sex)) + 
  geom_bar(position="fill") + 
  theme(axis.text.x = element_text(angle = 90)) + 
  ggtitle("relationship")

Vamos então balancear o gênero agrupando as categorias Wife e Husband, criando a categoria Married.

3.2.3 Capital Gain and Loss


ggplot(adult, aes(x= capital_gain, y= capital_loss)) +
  geom_point()
sum(adult$capital_loss > 0 & adult$capital_gain > 0)

Sendo assim, podemos soma-las e criar a variável capital_total sem medo de perder informação.

3.2.4 Worclass


ggplot(adult, aes(x = workclass)) +
  geom_bar() +
  theme(axis.text.x = element_text(angle = 90)) + 
  ggtitle("Workclass")

ggplot(adult, aes(x = workclass, fill = resposta)) + 
  geom_bar(position="fill") + 
  theme(axis.text.x = element_text(angle = 90)) + 
  ggtitle("Workclass")

Pelo visto a catgoria NA possui relação com a variável resposta distinta de todas as outras categorias, vamos então gerar uma nova categoria not-identify para atribuir os valores NA.

3.2.5 native_country


med <- (adult %>% 
          select(resposta) %>%
          filter(resposta == ">50K") %>% 
          count() %>% 
          as.numeric())/nrow(adult)
         

tb_country<- adult %>% 
                select(native_country, resposta) %>% 
                group_by(native_country) %>% 
                count(resposta) %>% 
                mutate(prop = prop.table(n)) %>% 
                filter(resposta == ">50K") %>% 
                mutate( class = case_when( native_country == "United-States" ~ "United-States",
                                           prop > med ~ ">mean",
                                           prop <= med ~ "<=mean" )  )

tb_country %>% 
  select(native_country,class) %>% 
  group_by(class) %>% 
  count()

Ficamos então com 21 países com proporções abaixo da méda, 18 acima e “United-States” como as 3 categorias restantes.


A distribuição ficou com 5% para países acima da média e 5% para países abaixo da média.

4 Modelagem

Com nossa a análise exploratória concluída, vamos dar início as estapas da modelagem utilizando o framework do tidymodels.

4.1 Amostragem

Fazendo a separação dos dados em treino e teste para a modelagem.

4.2 Data Prep

Os tratamentos necessários observados na AED, que foi feita utilizando o pacote DataExplorer e a função AED_biv que gerei para entender o comportamento das variáveis com relação a variável resposta, serão armazenados utilizando o recipes para ser utilizado tanto para treinar os modelos como para testar posteriormente.

4.3 Cross-Validation

Especificando a validação cruzada:

set.seed(32)
adult_vfold <- vfold_cv(adult_train, v = 5, strata = resposta)
adult_vfold
#  5-fold cross-validation using stratification 

4.4 Modelos

Os modelos que serão ajustados:

  • Decision tree
  • Random Forest
  • Xgboost

4.4.1 Decision tree

Especificando modelo:

adult_tree
Decision Tree Model Specification (classification)

Main Arguments:
  cost_complexity = tune()
  tree_depth = tune()
  min_n = tune()

Computational engine: rpart 

Workflow para decision tree:

Parâmentros:

hiperparams
Collection of 3 parameters for tuning

              id  parameter type object class
 cost_complexity cost_complexity    nparam[+]
      tree_depth      tree_depth    nparam[+]
           min_n           min_n    nparam[+]

Grid:

Efetuando tunagem de hiperparâmetros:

tree_tune <- 
  workflow_adult_tree %>% 
  tune_grid(
    resamples = adult_vfold,
    grid = tree_grid,
    control = control_grid(save_pred = TRUE, verbose = T, allow_par = F),
    metrics = metric_set(roc_auc)
  )
i Fold1: recipe
v Fold1: recipe
i Fold1: model  1/10
v Fold1: model  1/10
i Fold1: model  1/10 (predictions)
i Fold1: model  2/10
v Fold1: model  2/10
i Fold1: model  2/10 (predictions)
i Fold1: model  3/10
v Fold1: model  3/10
i Fold1: model  3/10 (predictions)
i Fold1: model  4/10
v Fold1: model  4/10
i Fold1: model  4/10 (predictions)
i Fold1: model  5/10
v Fold1: model  5/10
i Fold1: model  5/10 (predictions)
i Fold1: model  6/10
v Fold1: model  6/10
i Fold1: model  6/10 (predictions)
i Fold1: model  7/10
v Fold1: model  7/10
i Fold1: model  7/10 (predictions)
i Fold1: model  8/10
v Fold1: model  8/10
i Fold1: model  8/10 (predictions)
i Fold1: model  9/10
v Fold1: model  9/10
i Fold1: model  9/10 (predictions)
i Fold1: model 10/10
v Fold1: model 10/10
i Fold1: model 10/10 (predictions)
i Fold2: recipe
v Fold2: recipe
i Fold2: model  1/10
v Fold2: model  1/10
i Fold2: model  1/10 (predictions)
i Fold2: model  2/10
v Fold2: model  2/10
i Fold2: model  2/10 (predictions)
i Fold2: model  3/10
v Fold2: model  3/10
i Fold2: model  3/10 (predictions)
i Fold2: model  4/10
v Fold2: model  4/10
i Fold2: model  4/10 (predictions)
i Fold2: model  5/10
v Fold2: model  5/10
i Fold2: model  5/10 (predictions)
i Fold2: model  6/10
v Fold2: model  6/10
i Fold2: model  6/10 (predictions)
i Fold2: model  7/10
v Fold2: model  7/10
i Fold2: model  7/10 (predictions)
i Fold2: model  8/10
v Fold2: model  8/10
i Fold2: model  8/10 (predictions)
i Fold2: model  9/10
v Fold2: model  9/10
i Fold2: model  9/10 (predictions)
i Fold2: model 10/10
v Fold2: model 10/10
i Fold2: model 10/10 (predictions)
i Fold3: recipe
v Fold3: recipe
i Fold3: model  1/10
v Fold3: model  1/10
i Fold3: model  1/10 (predictions)
i Fold3: model  2/10
v Fold3: model  2/10
i Fold3: model  2/10 (predictions)
i Fold3: model  3/10
v Fold3: model  3/10
i Fold3: model  3/10 (predictions)
i Fold3: model  4/10
v Fold3: model  4/10
i Fold3: model  4/10 (predictions)
i Fold3: model  5/10
v Fold3: model  5/10
i Fold3: model  5/10 (predictions)
i Fold3: model  6/10
v Fold3: model  6/10
i Fold3: model  6/10 (predictions)
i Fold3: model  7/10
v Fold3: model  7/10
i Fold3: model  7/10 (predictions)
i Fold3: model  8/10
v Fold3: model  8/10
i Fold3: model  8/10 (predictions)
i Fold3: model  9/10
v Fold3: model  9/10
i Fold3: model  9/10 (predictions)
i Fold3: model 10/10
v Fold3: model 10/10
i Fold3: model 10/10 (predictions)
i Fold4: recipe
v Fold4: recipe
i Fold4: model  1/10
v Fold4: model  1/10
i Fold4: model  1/10 (predictions)
i Fold4: model  2/10
v Fold4: model  2/10
i Fold4: model  2/10 (predictions)
i Fold4: model  3/10
v Fold4: model  3/10
i Fold4: model  3/10 (predictions)
i Fold4: model  4/10
v Fold4: model  4/10
i Fold4: model  4/10 (predictions)
i Fold4: model  5/10
v Fold4: model  5/10
i Fold4: model  5/10 (predictions)
i Fold4: model  6/10
v Fold4: model  6/10
i Fold4: model  6/10 (predictions)
i Fold4: model  7/10
v Fold4: model  7/10
i Fold4: model  7/10 (predictions)
i Fold4: model  8/10
v Fold4: model  8/10
i Fold4: model  8/10 (predictions)
i Fold4: model  9/10
v Fold4: model  9/10
i Fold4: model  9/10 (predictions)
i Fold4: model 10/10
v Fold4: model 10/10
i Fold4: model 10/10 (predictions)
i Fold5: recipe
v Fold5: recipe
i Fold5: model  1/10
v Fold5: model  1/10
i Fold5: model  1/10 (predictions)
i Fold5: model  2/10
v Fold5: model  2/10
i Fold5: model  2/10 (predictions)
i Fold5: model  3/10
v Fold5: model  3/10
i Fold5: model  3/10 (predictions)
i Fold5: model  4/10
v Fold5: model  4/10
i Fold5: model  4/10 (predictions)
i Fold5: model  5/10
v Fold5: model  5/10
i Fold5: model  5/10 (predictions)
i Fold5: model  6/10
v Fold5: model  6/10
i Fold5: model  6/10 (predictions)
i Fold5: model  7/10
v Fold5: model  7/10
i Fold5: model  7/10 (predictions)
i Fold5: model  8/10
v Fold5: model  8/10
i Fold5: model  8/10 (predictions)
i Fold5: model  9/10
v Fold5: model  9/10
i Fold5: model  9/10 (predictions)
i Fold5: model 10/10
v Fold5: model 10/10
i Fold5: model 10/10 (predictions)

Finalizando WF:

workflow_tree_final
== Workflow ====================================================================
Preprocessor: Recipe
Model: decision_tree()

-- Preprocessor ----------------------------------------------------------------
7 Recipe Steps

* step_mutate()
* step_rm()
* step_string2factor()
* step_normalize()
* step_zv()
* step_novel()
* step_dummy()

-- Model -----------------------------------------------------------------------
Decision Tree Model Specification (classification)

Main Arguments:
  cost_complexity = 1.17576363081513e-05
  tree_depth = 8
  min_n = 13

Computational engine: rpart 

Verificando importância dos atributos:

Modelo final:

4.4.2 Random Forest

Especificando modelo:

adult_rf
Random Forest Model Specification (classification)

Main Arguments:
  mtry = tune()
  trees = tune()
  min_n = tune()

Computational engine: randomForest 

Workflow para random forest:

Grid:

Efetuando tunagem de hiperparâmetros:

set.seed(123)
rf_tune<- 
  workflow_adult_rf %>% 
  tune_grid(
    resamples = adult_vfold,
    grid = rf_grid,
    control = control_grid(save_pred = TRUE, verbose = T, allow_par = T),
    metrics = metric_set(roc_auc)
  )
i Fold1: recipe
v Fold1: recipe
i Fold1: model  1/10
v Fold1: model  1/10
i Fold1: model  1/10 (predictions)
i Fold1: model  2/10
v Fold1: model  2/10
i Fold1: model  2/10 (predictions)
i Fold1: model  3/10
v Fold1: model  3/10
i Fold1: model  3/10 (predictions)
i Fold1: model  4/10
v Fold1: model  4/10
i Fold1: model  4/10 (predictions)
i Fold1: model  5/10
v Fold1: model  5/10
i Fold1: model  5/10 (predictions)
i Fold1: model  6/10
v Fold1: model  6/10
i Fold1: model  6/10 (predictions)
i Fold1: model  7/10
v Fold1: model  7/10
i Fold1: model  7/10 (predictions)
i Fold1: model  8/10
v Fold1: model  8/10
i Fold1: model  8/10 (predictions)
i Fold1: model  9/10
v Fold1: model  9/10
i Fold1: model  9/10 (predictions)
i Fold1: model 10/10
v Fold1: model 10/10
i Fold1: model 10/10 (predictions)
i Fold2: recipe
v Fold2: recipe
i Fold2: model  1/10
v Fold2: model  1/10
i Fold2: model  1/10 (predictions)
i Fold2: model  2/10
v Fold2: model  2/10
i Fold2: model  2/10 (predictions)
i Fold2: model  3/10
v Fold2: model  3/10
i Fold2: model  3/10 (predictions)
i Fold2: model  4/10
v Fold2: model  4/10
i Fold2: model  4/10 (predictions)
i Fold2: model  5/10
v Fold2: model  5/10
i Fold2: model  5/10 (predictions)
i Fold2: model  6/10
v Fold2: model  6/10
i Fold2: model  6/10 (predictions)
i Fold2: model  7/10
v Fold2: model  7/10
i Fold2: model  7/10 (predictions)
i Fold2: model  8/10
v Fold2: model  8/10
i Fold2: model  8/10 (predictions)
i Fold2: model  9/10
v Fold2: model  9/10
i Fold2: model  9/10 (predictions)
i Fold2: model 10/10
v Fold2: model 10/10
i Fold2: model 10/10 (predictions)
i Fold3: recipe
v Fold3: recipe
i Fold3: model  1/10
v Fold3: model  1/10
i Fold3: model  1/10 (predictions)
i Fold3: model  2/10
v Fold3: model  2/10
i Fold3: model  2/10 (predictions)
i Fold3: model  3/10
v Fold3: model  3/10
i Fold3: model  3/10 (predictions)
i Fold3: model  4/10
v Fold3: model  4/10
i Fold3: model  4/10 (predictions)
i Fold3: model  5/10
v Fold3: model  5/10
i Fold3: model  5/10 (predictions)
i Fold3: model  6/10
v Fold3: model  6/10
i Fold3: model  6/10 (predictions)
i Fold3: model  7/10
v Fold3: model  7/10
i Fold3: model  7/10 (predictions)
i Fold3: model  8/10
v Fold3: model  8/10
i Fold3: model  8/10 (predictions)
i Fold3: model  9/10
v Fold3: model  9/10
i Fold3: model  9/10 (predictions)
i Fold3: model 10/10
v Fold3: model 10/10
i Fold3: model 10/10 (predictions)
i Fold4: recipe
v Fold4: recipe
i Fold4: model  1/10
v Fold4: model  1/10
i Fold4: model  1/10 (predictions)
i Fold4: model  2/10
v Fold4: model  2/10
i Fold4: model  2/10 (predictions)
i Fold4: model  3/10
v Fold4: model  3/10
i Fold4: model  3/10 (predictions)
i Fold4: model  4/10
v Fold4: model  4/10
i Fold4: model  4/10 (predictions)
i Fold4: model  5/10
v Fold4: model  5/10
i Fold4: model  5/10 (predictions)
i Fold4: model  6/10
v Fold4: model  6/10
i Fold4: model  6/10 (predictions)
i Fold4: model  7/10
v Fold4: model  7/10
i Fold4: model  7/10 (predictions)
i Fold4: model  8/10
v Fold4: model  8/10
i Fold4: model  8/10 (predictions)
i Fold4: model  9/10
v Fold4: model  9/10
i Fold4: model  9/10 (predictions)
i Fold4: model 10/10
v Fold4: model 10/10
i Fold4: model 10/10 (predictions)
i Fold5: recipe
v Fold5: recipe
i Fold5: model  1/10
v Fold5: model  1/10
i Fold5: model  1/10 (predictions)
i Fold5: model  2/10
v Fold5: model  2/10
i Fold5: model  2/10 (predictions)
i Fold5: model  3/10
v Fold5: model  3/10
i Fold5: model  3/10 (predictions)
i Fold5: model  4/10
v Fold5: model  4/10
i Fold5: model  4/10 (predictions)
i Fold5: model  5/10
v Fold5: model  5/10
i Fold5: model  5/10 (predictions)
i Fold5: model  6/10
v Fold5: model  6/10
i Fold5: model  6/10 (predictions)
i Fold5: model  7/10
v Fold5: model  7/10
i Fold5: model  7/10 (predictions)
i Fold5: model  8/10
v Fold5: model  8/10
i Fold5: model  8/10 (predictions)
i Fold5: model  9/10
v Fold5: model  9/10
i Fold5: model  9/10 (predictions)
i Fold5: model 10/10
v Fold5: model 10/10
i Fold5: model 10/10 (predictions)

Finalizando WF:

workflow_rf_final
== Workflow ====================================================================
Preprocessor: Recipe
Model: rand_forest()

-- Preprocessor ----------------------------------------------------------------
7 Recipe Steps

* step_mutate()
* step_rm()
* step_string2factor()
* step_normalize()
* step_zv()
* step_novel()
* step_dummy()

-- Model -----------------------------------------------------------------------
Random Forest Model Specification (classification)

Main Arguments:
  mtry = 42
  trees = 1406
  min_n = 21

Computational engine: randomForest 

Verificando importância dos atributos:

workflow_rf_final %>%
  fit(adult_train) %>%
  pull_workflow_fit() %>%
  vip::vip(geom = "col")

Modelo final:


rf_final <- last_fit(workflow_rf_final, adult_split)
collect_metrics(rf_final) #roc_auc = 0.9072885

4.4.3 Xgboost

Tunando mtry, trees e sample size:

adult_xgb
Boosted Tree Model Specification (classification)

Main Arguments:
  mtry = tune()
  trees = tune()
  min_n = tune()
  tree_depth = tune()
  learn_rate = tune()

Computational engine: xgboost 

Workflow para Xgboost:

Grid:

Efetuando tunagem de hiperparâmetros:

ini <- Sys.time()
xgb_tune <-
  workflow_adult_xgb %>%
    tune_grid(
        resamples = adult_vfold,
        grid = xgb_grid,
        control = control_grid(verbose = TRUE),
        metrics = metric_set(roc_auc)
    )
i Fold1: recipe
v Fold1: recipe
i Fold1: model  1/20
v Fold1: model  1/20
i Fold1: model  1/20 (predictions)
i Fold1: model  2/20
v Fold1: model  2/20
i Fold1: model  2/20 (predictions)
i Fold1: model  3/20
v Fold1: model  3/20
i Fold1: model  3/20 (predictions)
i Fold1: model  4/20
v Fold1: model  4/20
i Fold1: model  4/20 (predictions)
i Fold1: model  5/20
v Fold1: model  5/20
i Fold1: model  5/20 (predictions)
i Fold1: model  6/20
v Fold1: model  6/20
i Fold1: model  6/20 (predictions)
i Fold1: model  7/20
v Fold1: model  7/20
i Fold1: model  7/20 (predictions)
i Fold1: model  8/20
v Fold1: model  8/20
i Fold1: model  8/20 (predictions)
i Fold1: model  9/20
v Fold1: model  9/20
i Fold1: model  9/20 (predictions)
i Fold1: model 10/20
v Fold1: model 10/20
i Fold1: model 10/20 (predictions)
i Fold1: model 11/20
v Fold1: model 11/20
i Fold1: model 11/20 (predictions)
i Fold1: model 12/20
v Fold1: model 12/20
i Fold1: model 12/20 (predictions)
i Fold1: model 13/20
v Fold1: model 13/20
i Fold1: model 13/20 (predictions)
i Fold1: model 14/20
v Fold1: model 14/20
i Fold1: model 14/20 (predictions)
i Fold1: model 15/20
v Fold1: model 15/20
i Fold1: model 15/20 (predictions)
i Fold1: model 16/20
v Fold1: model 16/20
i Fold1: model 16/20 (predictions)
i Fold1: model 17/20
v Fold1: model 17/20
i Fold1: model 17/20 (predictions)
i Fold1: model 18/20
v Fold1: model 18/20
i Fold1: model 18/20 (predictions)
i Fold1: model 19/20
v Fold1: model 19/20
i Fold1: model 19/20 (predictions)
i Fold1: model 20/20
v Fold1: model 20/20
i Fold1: model 20/20 (predictions)
i Fold2: recipe
v Fold2: recipe
i Fold2: model  1/20
v Fold2: model  1/20
i Fold2: model  1/20 (predictions)
i Fold2: model  2/20
v Fold2: model  2/20
i Fold2: model  2/20 (predictions)
i Fold2: model  3/20
v Fold2: model  3/20
i Fold2: model  3/20 (predictions)
i Fold2: model  4/20
v Fold2: model  4/20
i Fold2: model  4/20 (predictions)
i Fold2: model  5/20
v Fold2: model  5/20
i Fold2: model  5/20 (predictions)
i Fold2: model  6/20
v Fold2: model  6/20
i Fold2: model  6/20 (predictions)
i Fold2: model  7/20
v Fold2: model  7/20
i Fold2: model  7/20 (predictions)
i Fold2: model  8/20
v Fold2: model  8/20
i Fold2: model  8/20 (predictions)
i Fold2: model  9/20
v Fold2: model  9/20
i Fold2: model  9/20 (predictions)
i Fold2: model 10/20
v Fold2: model 10/20
i Fold2: model 10/20 (predictions)
i Fold2: model 11/20
v Fold2: model 11/20
i Fold2: model 11/20 (predictions)
i Fold2: model 12/20
v Fold2: model 12/20
i Fold2: model 12/20 (predictions)
i Fold2: model 13/20
v Fold2: model 13/20
i Fold2: model 13/20 (predictions)
i Fold2: model 14/20
v Fold2: model 14/20
i Fold2: model 14/20 (predictions)
i Fold2: model 15/20
v Fold2: model 15/20
i Fold2: model 15/20 (predictions)
i Fold2: model 16/20
v Fold2: model 16/20
i Fold2: model 16/20 (predictions)
i Fold2: model 17/20
v Fold2: model 17/20
i Fold2: model 17/20 (predictions)
i Fold2: model 18/20
v Fold2: model 18/20
i Fold2: model 18/20 (predictions)
i Fold2: model 19/20
v Fold2: model 19/20
i Fold2: model 19/20 (predictions)
i Fold2: model 20/20
v Fold2: model 20/20
i Fold2: model 20/20 (predictions)
i Fold3: recipe
v Fold3: recipe
i Fold3: model  1/20
v Fold3: model  1/20
i Fold3: model  1/20 (predictions)
i Fold3: model  2/20
v Fold3: model  2/20
i Fold3: model  2/20 (predictions)
i Fold3: model  3/20
v Fold3: model  3/20
i Fold3: model  3/20 (predictions)
i Fold3: model  4/20
v Fold3: model  4/20
i Fold3: model  4/20 (predictions)
i Fold3: model  5/20
v Fold3: model  5/20
i Fold3: model  5/20 (predictions)
i Fold3: model  6/20
v Fold3: model  6/20
i Fold3: model  6/20 (predictions)
i Fold3: model  7/20
v Fold3: model  7/20
i Fold3: model  7/20 (predictions)
i Fold3: model  8/20
v Fold3: model  8/20
i Fold3: model  8/20 (predictions)
i Fold3: model  9/20
v Fold3: model  9/20
i Fold3: model  9/20 (predictions)
i Fold3: model 10/20
v Fold3: model 10/20
i Fold3: model 10/20 (predictions)
i Fold3: model 11/20
v Fold3: model 11/20
i Fold3: model 11/20 (predictions)
i Fold3: model 12/20
v Fold3: model 12/20
i Fold3: model 12/20 (predictions)
i Fold3: model 13/20
v Fold3: model 13/20
i Fold3: model 13/20 (predictions)
i Fold3: model 14/20
v Fold3: model 14/20
i Fold3: model 14/20 (predictions)
i Fold3: model 15/20
v Fold3: model 15/20
i Fold3: model 15/20 (predictions)
i Fold3: model 16/20
v Fold3: model 16/20
i Fold3: model 16/20 (predictions)
i Fold3: model 17/20
v Fold3: model 17/20
i Fold3: model 17/20 (predictions)
i Fold3: model 18/20
v Fold3: model 18/20
i Fold3: model 18/20 (predictions)
i Fold3: model 19/20
v Fold3: model 19/20
i Fold3: model 19/20 (predictions)
i Fold3: model 20/20
v Fold3: model 20/20
i Fold3: model 20/20 (predictions)
i Fold4: recipe
v Fold4: recipe
i Fold4: model  1/20
v Fold4: model  1/20
i Fold4: model  1/20 (predictions)
i Fold4: model  2/20
v Fold4: model  2/20
i Fold4: model  2/20 (predictions)
i Fold4: model  3/20
v Fold4: model  3/20
i Fold4: model  3/20 (predictions)
i Fold4: model  4/20
v Fold4: model  4/20
i Fold4: model  4/20 (predictions)
i Fold4: model  5/20
v Fold4: model  5/20
i Fold4: model  5/20 (predictions)
i Fold4: model  6/20
v Fold4: model  6/20
i Fold4: model  6/20 (predictions)
i Fold4: model  7/20
v Fold4: model  7/20
i Fold4: model  7/20 (predictions)
i Fold4: model  8/20
v Fold4: model  8/20
i Fold4: model  8/20 (predictions)
i Fold4: model  9/20
v Fold4: model  9/20
i Fold4: model  9/20 (predictions)
i Fold4: model 10/20
v Fold4: model 10/20
i Fold4: model 10/20 (predictions)
i Fold4: model 11/20
v Fold4: model 11/20
i Fold4: model 11/20 (predictions)
i Fold4: model 12/20
v Fold4: model 12/20
i Fold4: model 12/20 (predictions)
i Fold4: model 13/20
v Fold4: model 13/20
i Fold4: model 13/20 (predictions)
i Fold4: model 14/20
v Fold4: model 14/20
i Fold4: model 14/20 (predictions)
i Fold4: model 15/20
v Fold4: model 15/20
i Fold4: model 15/20 (predictions)
i Fold4: model 16/20
v Fold4: model 16/20
i Fold4: model 16/20 (predictions)
i Fold4: model 17/20
v Fold4: model 17/20
i Fold4: model 17/20 (predictions)
i Fold4: model 18/20
v Fold4: model 18/20
i Fold4: model 18/20 (predictions)
i Fold4: model 19/20
v Fold4: model 19/20
i Fold4: model 19/20 (predictions)
i Fold4: model 20/20
v Fold4: model 20/20
i Fold4: model 20/20 (predictions)
i Fold5: recipe
v Fold5: recipe
i Fold5: model  1/20
v Fold5: model  1/20
i Fold5: model  1/20 (predictions)
i Fold5: model  2/20
v Fold5: model  2/20
i Fold5: model  2/20 (predictions)
i Fold5: model  3/20
v Fold5: model  3/20
i Fold5: model  3/20 (predictions)
i Fold5: model  4/20
v Fold5: model  4/20
i Fold5: model  4/20 (predictions)
i Fold5: model  5/20
v Fold5: model  5/20
i Fold5: model  5/20 (predictions)
i Fold5: model  6/20
v Fold5: model  6/20
i Fold5: model  6/20 (predictions)
i Fold5: model  7/20
v Fold5: model  7/20
i Fold5: model  7/20 (predictions)
i Fold5: model  8/20
v Fold5: model  8/20
i Fold5: model  8/20 (predictions)
i Fold5: model  9/20
v Fold5: model  9/20
i Fold5: model  9/20 (predictions)
i Fold5: model 10/20
v Fold5: model 10/20
i Fold5: model 10/20 (predictions)
i Fold5: model 11/20
v Fold5: model 11/20
i Fold5: model 11/20 (predictions)
i Fold5: model 12/20
v Fold5: model 12/20
i Fold5: model 12/20 (predictions)
i Fold5: model 13/20
v Fold5: model 13/20
i Fold5: model 13/20 (predictions)
i Fold5: model 14/20
v Fold5: model 14/20
i Fold5: model 14/20 (predictions)
i Fold5: model 15/20
v Fold5: model 15/20
i Fold5: model 15/20 (predictions)
i Fold5: model 16/20
v Fold5: model 16/20
i Fold5: model 16/20 (predictions)
i Fold5: model 17/20
v Fold5: model 17/20
i Fold5: model 17/20 (predictions)
i Fold5: model 18/20
v Fold5: model 18/20
i Fold5: model 18/20 (predictions)
i Fold5: model 19/20
v Fold5: model 19/20
i Fold5: model 19/20 (predictions)
i Fold5: model 20/20
v Fold5: model 20/20
i Fold5: model 20/20 (predictions)
Sys.time()- ini #Time difference of 1.034167 hours ( 35 mins with parallel)
Time difference of 39.9844 mins

Finalizando WF:

workflow_xgb_final
== Workflow ====================================================================
Preprocessor: Recipe
Model: boost_tree()

-- Preprocessor ----------------------------------------------------------------
7 Recipe Steps

* step_mutate()
* step_rm()
* step_string2factor()
* step_normalize()
* step_zv()
* step_novel()
* step_dummy()

-- Model -----------------------------------------------------------------------
Boosted Tree Model Specification (classification)

Main Arguments:
  mtry = 34
  trees = 1309
  min_n = 5
  tree_depth = 10
  learn_rate = 0.0106445048353615

Computational engine: xgboost 

Verificando importância dos atributos:

workflow_xgb_final %>%
  fit(adult_train) %>%
  pull_workflow_fit() %>%
  vip::vip(geom = "col")

Modelo final:

4.4.4 Xgboost2

Tunando mtry, trees e sample size:

adult_xgb2
Boosted Tree Model Specification (classification)

Main Arguments:
  mtry = 34
  trees = 1309
  min_n = 5
  tree_depth = 10
  learn_rate = 0.0106445
  loss_reduction = tune()
  sample_size = tune()

Computational engine: xgboost 

Workflow para Xgboost:

workflow_adult_xgb2
== Workflow ====================================================================
Preprocessor: Recipe
Model: boost_tree()

-- Preprocessor ----------------------------------------------------------------
7 Recipe Steps

* step_mutate()
* step_rm()
* step_string2factor()
* step_normalize()
* step_zv()
* step_novel()
* step_dummy()

-- Model -----------------------------------------------------------------------
Boosted Tree Model Specification (classification)

Main Arguments:
  mtry = 34
  trees = 1309
  min_n = 5
  tree_depth = 10
  learn_rate = 0.0106445
  loss_reduction = tune()
  sample_size = tune()

Computational engine: xgboost 

Grid:

Efetuando tunagem de hiperparâmetros:

getDoParWorkers()
[1] 3

Finalizando WF:

workflow_xgb_final2
== Workflow ====================================================================
Preprocessor: Recipe
Model: boost_tree()

-- Preprocessor ----------------------------------------------------------------
7 Recipe Steps

* step_mutate()
* step_rm()
* step_string2factor()
* step_normalize()
* step_zv()
* step_novel()
* step_dummy()

-- Model -----------------------------------------------------------------------
Boosted Tree Model Specification (classification)

Main Arguments:
  mtry = 34
  trees = 1309
  min_n = 5
  tree_depth = 10
  learn_rate = 0.0106445
  loss_reduction = 2.08764734275328
  sample_size = 0.869696747139096

Computational engine: xgboost 

Verificando importância dos atributos:

workflow_xgb_final2 %>%
  fit(adult_train) %>%
  pull_workflow_fit() %>%
  vip::vip(geom = "col")

Modelo final:

4.4.5 Xgboost6

Tunando mtry, trees e sample size:

adult_xgb6
Boosted Tree Model Specification (classification)

Main Arguments:
  mtry = 33
  trees = 1347
  min_n = 5
  tree_depth = 7
  learn_rate = 0.01566693

Computational engine: xgboost 

Workflow para Xgboost:

Finalizando WF:

workflow_xgb_final6
== Workflow ====================================================================
Preprocessor: Recipe
Model: boost_tree()

-- Preprocessor ----------------------------------------------------------------
7 Recipe Steps

* step_mutate()
* step_rm()
* step_string2factor()
* step_normalize()
* step_zv()
* step_novel()
* step_dummy()

-- Model -----------------------------------------------------------------------
Boosted Tree Model Specification (classification)

Main Arguments:
  mtry = 33
  trees = 1347
  min_n = 5
  tree_depth = 7
  learn_rate = 0.01566693

Computational engine: xgboost 

Verificando importância dos atributos:

workflow_xgb_final6 %>%
  fit(adult_train) %>%
  pull_workflow_fit() %>%
  vip::vip(geom = "col")

Modelo final:

5 Scoragem para submeter resultado

Matriz de confusão:

adult_val %>% 
  transmute(resposta = factor(resposta, levels = c(">50K", "<=50K")), 
            more_than_50k = ifelse(more_than_50k > 0.5, ">50K", "<=50K") %>% 
              factor(levels = c(">50K", "<=50K"))) %>% 
  table() %>% 
  caret::confusionMatrix()
Confusion Matrix and Statistics

        more_than_50k
resposta  >50K <=50K
   >50K   2512  1334
   <=50K   726 11709
                                          
               Accuracy : 0.8735          
                 95% CI : (0.8683, 0.8785)
    No Information Rate : 0.8011          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6291          
                                          
 Mcnemar's Test P-Value : < 2.2e-16       
                                          
            Sensitivity : 0.7758          
            Specificity : 0.8977          
         Pos Pred Value : 0.6531          
         Neg Pred Value : 0.9416          
             Prevalence : 0.1989          
         Detection Rate : 0.1543          
   Detection Prevalence : 0.2362          
      Balanced Accuracy : 0.8368          
                                          
       'Positive' Class : >50K            
                                          
LS0tDQp0aXRsZTogIkRlc2FmaW8gSW50cm8gTUwgLSBDdXJzby1SIg0KYXV0aG9yOiAiUmljYXJkbyBNYXR0b3MiDQpkYXRlOiAiMTIvMDcvMjAyMCINCm91dHB1dDoNCiAgaHRtbF9ub3RlYm9vazoNCiAgICB0b2M6IHllcw0KICAgIHRvY19mbG9hdDogeWVzDQogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMNCiAgaHRtbF9kb2N1bWVudDoNCiAgICB0b2M6IHllcw0KICAgIHRvY19mbG9hdDogeWVzDQogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMNCi0tLQ0KDQoNCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQ0KbGlicmFyeShyZWFkcikNCmxpYnJhcnkodGlkeW1vZGVscykNCmxpYnJhcnkoZ2dwbG90MikNCmxpYnJhcnkoc2tpbXIpDQpsaWJyYXJ5KFJDdXJsKQ0KbGlicmFyeShrYWJsZUV4dHJhKQ0KbGlicmFyeShncmlkRXh0cmEpDQpsaWJyYXJ5KGdsdWUpDQpsaWJyYXJ5KGZvcmNhdHMpDQpsaWJyYXJ5KERhdGFFeHBsb3JlcikNCmBgYA0KDQojIE9iamV0aXZvDQoNCk8gb2JqZXRpdm8gZGVzc2Ugbm90ZWJvb2sgw6kgZWZldHVhciB0b2RvIG8gcHJvY2Vzc28gZGUgbW9kZWxhZ2VtIGRhIGJhc2UgZGUgZGFkb3MgYGFkdWx0YCwgZGlzcG9uaWJpbGl6YWRhIHBhcmEgbyBkZXNhZmlvIGRvIGN1cnNvIGRlIGludHJvZHXDp8OjbyBhbyBNYWNoaW5lIExlYXJuaW5nIGRhIEN1cnNvLVIsIHV0aWxpemFuZG8gbyBmcmFtZXdvcmsgYHRpZHltb2RlbHNgLiBPdSBzZWphLCBleHBsb3JhciwgdHJhdGFyLCBwcmVwYXJhciwgdHVubmFyIGUgZXNjb2xoZXIgbyBtb2RlbG8gcXVlIG1lbGhvciBzZSBhanVzdGEgYW9zIGRhZG9zIGRpc3BvbmliaWxpemFkb3MuIFZhbW9zIG5lc3NhIQ0KDQoNCiMgTGVpdHVyYSBkYSBiYXNlDQoNCiMjIEluZm9ybWHDp8O1ZXMgcHJlbGltaW5hcmVzDQoNCmBgYHtyfQ0KYWR1bHQgPC0gcmVhZF9yZHMoImFkdWx0LnJkcyIpDQoNCiMgaGVhZChhZHVsdCkgDQoNCiMgZ2xpbXBzZShhZHVsdCkNCnNraW0oYWR1bHQpDQoNCmBgYA0KDQoNCg0KPGJyPiBBcyB2YXJpw6F2ZWlzIHBhcmVjZW0gZXN0YXIgY29tIGZvcm1hdG9zIGNvcnJldG9zLiBQb250byBkZSBhdGVuw6fDo28gcGFyYSBhcyB2YXJpw6F2ZWlzIGB3b2tjbGFzc2AsIGBvY2N1cGF0aW9uYCBlIGBuYXRpdmVfY291bnRyeWAsIHF1ZSBhcHJlc2VudGFtIHZhbG9yZXMgbWlzc2luZy4gIDwvYnI+DQoNCg0KIyBBRUQgDQoNCjxicj4gQWdvcmEgdmFtb3MgYW5hbGlzYXIgbyBjb21wb3J0YW1lbnRvIGRhcyB2YXJpw6F2ZWlzIHBhcmEgZGVmaW5pcm1vcyBjb21vIHRyYXRhciBvcyBub3Nzb3MgZGFkb3MgcGFyYSBvIG1vZGVsby4gPC9icj4NCg0KIyMgQW7DoWxpc2UgYml2YXJpYWRhIHsudGFic2V0fQ0KDQpgYGB7cixyZXN1bHRzPSdhc2lzJywgZWNobz1UUlVFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KDQoNCiMgRGF0YUV4cGxvcmVyOjpjcmVhdGVfcmVwb3J0KGFkdWx0KQ0KDQpkZXZ0b29sczo6c291cmNlX3VybCgiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL3JpY2FyZG9tYXR0b3MwNS9mdW5jdGlvbnMvbWFzdGVyL2Z1bmN0aW9uX0FFRF9iaXZhcmlhZGEuUiIpDQojIA0KIyANCmFkdWx0MiA8LSBhZHVsdCAlPiUNCiAgICAgICAgICAgIHNlbGVjdCgtaWQpICU+JSANCiAgICAgICAgICAgIG11dGF0ZShyZXNwb3N0YSA9IGlmX2Vsc2UocmVzcG9zdGEgPT0gIj41MEsiLCAxLCAwKSkNCiMgDQojIA0KDQojIG5hbWVzKGFkdWx0MikNCmZvciAoaSBpbiAxOihsZW5ndGgoYWR1bHQyKS0xKSApIHsNCiAgDQogIGRmIDwtIGFkdWx0MlssYyhpLDE1KV0NCiAgY2F0KCIjIyMgIixuYW1lcyhkZlssMV0pLCJcbiIpIA0KICBwcmludChBRURfYml2KGRmLGdsdWUoInJlc3Bvc3RhIiksIlByZSIpKQ0KICBjYXQoJ1xuXG4nKQ0KfQ0KDQoNCg0KYGBgDQoNCg0KIyMgey19DQoNCk9ic2VydmHDp8O1ZXM6DQoNCiogYGVkdWNhdGlvbmAgOiDDqSBwb3Nzw612ZWwgdmlzdWFsaXphciBxdWUgcXVhbnRvIG1haW9yIG8gZ3JhdSBkZSBlc2NvbGFyaWRhZGUsIG1haW9yIGEgcHJvcG9yw6fDo28gZGUgcGVzc29hcyBjb20gc2FsYXJpb3MgYWNpbWEgZGUgNTBrLiBFIHF1ZSBhcyBjYXRlZ29yaWFzIGFiYWl4byBkZSBIUy1ncmFkLCBgMXRoLTR0aGAgYXTDqSBgMTJ0aGBhbMOpbSBkZSBzZXJlbSBwb3VjbyByZXByZXNlbnRhdGl2YXMsIHBvc3N1ZW0gYmFpeGEgcHJvcG9yw6fDo28sIHZhbW9zIGVudMOjbyBjcmlhciB1bWEgY2F0ZWdvcmlhIHVtYSBub3ZhIGNvbnNvbGlkYW5kbyBlbGFzIGBIUy1ub3QtZ3JhZGAuDQoNCiogYG1hcml0YWxfc3RhdHVzYCA6IGFxdWkgaXJlbW9zIGFncnVwYXIgb3MgY2FtcG9zIGBNYXJyaWVkLUFGLXNwb3VzZWAgZSBgTWFycmllZC1jaXYtc3BvdXNlYCwgY3JpYW5kbyBhIGNhdGVnb3JpYSBgTWFycmllZGAsIGJhc2VhZG8gbmEgc2ltaWxhcmlkYWRlIGVudHJlIGVsYXMgY29tIHJlbGHDp8OjbyBhIHZhcmnDoXZlbCByZXNwb3N0YSBlIGNvbnNpZGVyYW5kbyBhIGRlc2NyacOnw6NvIGRlbGFzLg0KDQoqIGBuYXRpdmVfY291bnRyeWAgOiDDiSB1bSBjYW1wbyBjb20gcG91Y2EgdmFyaWFiaWxpZGFkZSwgb25kZSBgciAoYWR1bHQgJT4lIHNlbGVjdChuYXRpdmVfY291bnRyeSkgJT4lIGZpbHRlcihuYXRpdmVfY291bnRyeSA9PSAiVW5pdGVkLVN0YXRlcyIpICU+JSBjb3VudCgpIC8gY291bnQoYWR1bHQpKSAlPiUgYXMubnVtZXJpYygpICU+JSBwZXJjZW50KClgIGRvcyBkYWRvcyBlc3TDo28gYXRyaWJ1w61kb3MgY29tbyAiRXN0YWRvcyBVbmlkb3MiLiBTZW5kbyBhc3NpbSwgcG9kZXJpYSBjb25zaWRlcmFyIGFwZW5hcyBFc3RhZG9zIFVuaWRvcyBlIGFncnVwYXIgbyByZXN0YW50ZSBjb21vIG91dHJvcywgbWFzIHZhbW9zIG1hbnRlciBvIG3DoXhpbW8gZGUgaW5mb3JtYcOnw6NvIGUgcmVkdXppciBhcyBjYXRlZ29yaWFzIHBhcmEgMywgYWdydXBhbmRvIHRvZG9zIG9zIHBhw61zZXMgcXVlIG9idGl2ZXJhbSBwcm9wb3LDp8OjbyBtYWlvciBxdWUgYSBtw6lkaWEsIG1hbnRlciBvIHZhbG9yIG1haXMgcmVwcmVzZW50YXRpdm8gZSB1bWEgY2F0ZWdvcmlhIGNvbSBvcyBwYcOtc2VzIGFiYWl4byBkYSBtw6lkaWEuDQoNCiogYHJlbGF0aW9uc2hpcGAgOiBjYW1wbyBjb250w6ltIG9zIGNhbXBvcyBgaHVzYmFuZGAgZSBgd2lmZWAsIGFwYXJlbnRlbWVudGUgcG9kZXJpYW1vcyBhZ3J1cGEtbG9zLCB2YW1vcyBhbmFsaXNhciBtYWlzIGFmdW5kby4NCg0KKiBgY2FwaXRhbF9sb3NzYCBlIGBjYXBpdGFsX2dhaW5gIDogQXBhcmVudGVtZW50ZSB0YW50byBxdWVtIGdhbmhhIHF1YW50byBxdWVtIHBlcmRlIGFsZ3VtIHZhbG9yIGFwcmVzZW50YW0gbWFpb3JlcyBwcm9iYWJpbGlkYWRlcyBkZSB0ZXIgc2FsYXJpbyA+NTBrLiBWYW1vcyBlbnTDo28gYXZhbGlhciBhIGNvcnJlbGHDp8OjbyBlbnRyZSBlbGFzLg0KDQoqIGB3b3JrY2xhc3NgIDogQ2F0ZWdvcmlhcyBjb20gYmFpeGEgcmVwcmVzZW50YXRpdmlkYWRlIGNvbW8gYE5ldmVyLXdvcmtlZGBlIGBXaXRob3V0LXBheWAgbsOjbyBwb3NzdWVtIGNsYXNzaWZpY2HDp8OjbyBjb20gYSByZXNwb3N0YSBkZSBpbnRlcmVzc2UgIj41MGsiLCB2YW1vcyBkYXIgdW0gem9vbSBuZXNzYSB2YXJpw6F2ZWwgZSBhbmFsaXNhciBvcyBOQSdzIHF1ZSBpZGVudGlmaWNhbW9zIHRhbWLDqW0uDQoNCiMjIEFFRCBmaW5hbDogey50YWJzZXR9DQoNCiMjIyBvY2N1cGF0aW9uDQoNCmBgYHtyfQ0KDQpnZ3Bsb3QoYWR1bHQsIGFlcyh4ID0gb2NjdXBhdGlvbiwgZmlsbCA9IHJlc3Bvc3RhKSkgKyANCiAgZ2VvbV9iYXIocG9zaXRpb249ImZpbGwiKSArIA0KICB0aGVtZShheGlzLnRleHQueCA9IGVsZW1lbnRfdGV4dChhbmdsZSA9IDkwKSkgKyANCiAgZ2d0aXRsZSgib2NjdXBhdGlvbiIpDQoNCmBgYA0KDQo8YnI+IMOJIHBvc3PDrXZlbCB2ZXIgcXVlIG7Do28gZmFyaWEgc2VudGlkbyBhdHJpYnVpciBvcyBOQXMgZGUgZm9ybWEgbW9kYWwsIHVtYSB2ZXogcXVlIG5vc3NvIG9iamV0aXZvIMOpIG9idGVyIG8gbWFpb3IgcG9kZXIgcHJlZGl0aXZvIHBvc3PDrXZlbCwgbG9nbywgbsOjbyBxdWVyZW1vcyBwZXJkZXIgaW5mb3JtYcOnw6NvLiBTZW5kbyBhc3NpbSwgbsOjbyB2YW1vcyBkaWx1aXIgb3MgTkFzIG5hIGNhdGVnb3JpYSBjb20gbWFpb3IgcmVwcmVzZW50YXRpdmlkYWRlIGBQcm9mLXNwZWNpYWx0eWAsIHZhbW9zIGF0cmlidWlyIMOgIHVtYSBjYXRlZ29yaWEgY29tIHByb3BvcsOnw7VlcyBzaW1pbGFyZXMgZSBxdWUgcG9zc3VpIHVtYSBib2EgcmVwcmVzZW50YXRpdmlkYWRlLCBgRmFybWluZy1maXNoaW5nYC4gPC9icj4NCg0KDQojIyMgcmVsYXRpb25zaGlwDQpgYGB7cn0NCg0KZ2dwbG90KGFkdWx0LCBhZXMoeCA9IHJlbGF0aW9uc2hpcCkpICsNCiAgZ2VvbV9iYXIoKSArDQogIHRoZW1lKGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KGFuZ2xlID0gOTApKSArIA0KICBnZ3RpdGxlKCJyZWxhdGlvbnNoaXAiKQ0KDQpnZ3Bsb3QoYWR1bHQsIGFlcyh4ID0gcmVsYXRpb25zaGlwLCBmaWxsID0gcmVzcG9zdGEpKSArIA0KICBnZW9tX2Jhcihwb3NpdGlvbj0iZmlsbCIpICsgDQogIHRoZW1lKGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KGFuZ2xlID0gOTApKSArIA0KICBnZ3RpdGxlKCJyZWxhdGlvbnNoaXAiKQ0KDQpnZ3Bsb3QoYWR1bHQsIGFlcyh4ID0gcmVsYXRpb25zaGlwLCBmaWxsID0gc2V4KSkgKyANCiAgZ2VvbV9iYXIocG9zaXRpb249ImZpbGwiKSArIA0KICB0aGVtZShheGlzLnRleHQueCA9IGVsZW1lbnRfdGV4dChhbmdsZSA9IDkwKSkgKyANCiAgZ2d0aXRsZSgicmVsYXRpb25zaGlwIikNCg0KYGBgDQoNClZhbW9zIGVudMOjbyBiYWxhbmNlYXIgbyBnw6puZXJvIGFncnVwYW5kbyBhcyBjYXRlZ29yaWFzIFdpZmUgZSBIdXNiYW5kLCBjcmlhbmRvIGEgY2F0ZWdvcmlhIGBNYXJyaWVkYC4NCg0KIyMjIENhcGl0YWwgR2FpbiBhbmQgTG9zcw0KDQoNCmBgYHtyLCBlY2hvID0gVFJVRX0NCg0KZ2dwbG90KGFkdWx0LCBhZXMoeD0gY2FwaXRhbF9nYWluLCB5PSBjYXBpdGFsX2xvc3MpKSArDQogIGdlb21fcG9pbnQoKQ0KYGBgDQoNCg0KDQpgYGB7cn0NCnN1bShhZHVsdCRjYXBpdGFsX2xvc3MgPiAwICYgYWR1bHQkY2FwaXRhbF9nYWluID4gMCkNCmBgYA0KU2VuZG8gYXNzaW0sIHBvZGVtb3Mgc29tYS1sYXMgZSBjcmlhciBhIHZhcmnDoXZlbCBgY2FwaXRhbF90b3RhbGAgc2VtIG1lZG8gZGUgcGVyZGVyIGluZm9ybWHDp8Ojby4NCg0KIyMjIFdvcmNsYXNzDQoNCmBgYHtyfQ0KDQpnZ3Bsb3QoYWR1bHQsIGFlcyh4ID0gd29ya2NsYXNzKSkgKw0KICBnZW9tX2JhcigpICsNCiAgdGhlbWUoYXhpcy50ZXh0LnggPSBlbGVtZW50X3RleHQoYW5nbGUgPSA5MCkpICsgDQogIGdndGl0bGUoIldvcmtjbGFzcyIpDQoNCmdncGxvdChhZHVsdCwgYWVzKHggPSB3b3JrY2xhc3MsIGZpbGwgPSByZXNwb3N0YSkpICsgDQogIGdlb21fYmFyKHBvc2l0aW9uPSJmaWxsIikgKyANCiAgdGhlbWUoYXhpcy50ZXh0LnggPSBlbGVtZW50X3RleHQoYW5nbGUgPSA5MCkpICsgDQogIGdndGl0bGUoIldvcmtjbGFzcyIpDQoNCmBgYA0KDQpQZWxvIHZpc3RvIGEgY2F0Z29yaWEgTkEgcG9zc3VpIHJlbGHDp8OjbyBjb20gYSB2YXJpw6F2ZWwgcmVzcG9zdGEgZGlzdGludGEgZGUgdG9kYXMgYXMgb3V0cmFzIGNhdGVnb3JpYXMsIHZhbW9zIGVudMOjbyBnZXJhciB1bWEgbm92YSBjYXRlZ29yaWEgYG5vdC1pZGVudGlmeWAgcGFyYSBhdHJpYnVpciBvcyB2YWxvcmVzIE5BLg0KDQoNCiMjIyBuYXRpdmVfY291bnRyeQ0KDQpgYGB7cn0NCg0KbWVkIDwtIChhZHVsdCAlPiUgDQogICAgICAgICAgc2VsZWN0KHJlc3Bvc3RhKSAlPiUNCiAgICAgICAgICBmaWx0ZXIocmVzcG9zdGEgPT0gIj41MEsiKSAlPiUgDQogICAgICAgICAgY291bnQoKSAlPiUgDQogICAgICAgICAgYXMubnVtZXJpYygpKS9ucm93KGFkdWx0KQ0KICAgICAgICAgDQoNCnRiX2NvdW50cnk8LSBhZHVsdCAlPiUgDQogICAgICAgICAgICAgICAgc2VsZWN0KG5hdGl2ZV9jb3VudHJ5LCByZXNwb3N0YSkgJT4lIA0KICAgICAgICAgICAgICAgIGdyb3VwX2J5KG5hdGl2ZV9jb3VudHJ5KSAlPiUgDQogICAgICAgICAgICAgICAgY291bnQocmVzcG9zdGEpICU+JSANCiAgICAgICAgICAgICAgICBtdXRhdGUocHJvcCA9IHByb3AudGFibGUobikpICU+JSANCiAgICAgICAgICAgICAgICBmaWx0ZXIocmVzcG9zdGEgPT0gIj41MEsiKSAlPiUgDQogICAgICAgICAgICAgICAgbXV0YXRlKCBjbGFzcyA9IGNhc2Vfd2hlbiggbmF0aXZlX2NvdW50cnkgPT0gIlVuaXRlZC1TdGF0ZXMiIH4gIlVuaXRlZC1TdGF0ZXMiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByb3AgPiBtZWQgfiAiPm1lYW4iLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByb3AgPD0gbWVkIH4gIjw9bWVhbiIgKSAgKQ0KDQp0Yl9jb3VudHJ5ICU+JSANCiAgc2VsZWN0KG5hdGl2ZV9jb3VudHJ5LGNsYXNzKSAlPiUgDQogIGdyb3VwX2J5KGNsYXNzKSAlPiUgDQogIGNvdW50KCkNCg0KDQoNCmBgYA0KDQoNCkZpY2Ftb3MgZW50w6NvIGNvbSAyMSBwYcOtc2VzIGNvbSBwcm9wb3LDp8O1ZXMgYWJhaXhvIGRhIG3DqWRhLCAxOCBhY2ltYSBlICJVbml0ZWQtU3RhdGVzIiBjb21vIGFzIDMgY2F0ZWdvcmlhcyByZXN0YW50ZXMuDQoNCg0KYGBge3IsIGVjaG89RkFMU0V9DQoNCiMgdGJfY291bnRyeSAlPiUNCiMgICAgICAgICBmaWx0ZXIoY2xhc3MgPT0gIjw9bWVhbiIpICU+JQ0KIyAgICAgICAgIHNlbGVjdChuYXRpdmVfY291bnRyeSkgJT4lDQojICAgICAgICAgYXMuZmFjdG9yKCkNCg0KDQphZHVsdDI8LSBhZHVsdDIgJT4lDQogICAgbXV0YXRlKGNsYXNzX2NvdW50cnkgPSBjYXNlX3doZW4obmF0aXZlX2NvdW50cnkgJWluJSBjKCJDYW1ib2RpYSIsICJDYW5hZGEiLCAiQ2hpbmEiLCAiQ3ViYSIsICJFbmdsYW5kIiwgIkZyYW5jZSIsICJHZXJtYW55IiwgIkdyZWVjZSIsICJIb25nIiwgIkluZGlhIiwgIklyYW4iLCAiSXRhbHkiLCAiSmFwYW4iLCAiUGhpbGlwcGluZXMiLCAiU2NvdGxhbmQiLCAiVGFpd2FuIiwgIll1Z29zbGF2aWEiLCBOQSkgIH4gIj5tZWFuIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbmF0aXZlX2NvdW50cnkgPT0gIlVuaXRlZC1TdGF0ZXMiIH4gIlVuaXRlZC1TdGF0ZXMiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICBUUlVFIH4gIjw9bWVhbiIpICkgICAgIA0KDQojIGFkdWx0MiAlPiUgDQojICAgZmlsdGVyKGNsYXNzID09ICI+bWVhbiIpICU+JSANCiMgICBzZWxlY3QobmF0aXZlX2NvdW50cnkpICU+JSANCiMgICBncm91cF9ieShuYXRpdmVfY291bnRyeSkgJT4lIA0KIyAgIGNvdW50KCkNCg0KZ2dwbG90KGFkdWx0MiwgYWVzKHggPSBjbGFzc19jb3VudHJ5KSkgKw0KICBnZW9tX2JhcihhZXMoeSA9ICguLmNvdW50Li4pL3N1bSguLmNvdW50Li4pKSkgKw0KICBnZW9tX3RleHQoc3RhdCA9ICJjb3VudCIsIA0KICAgICAgICAgICAgYWVzKGxhYmVsID0gcm91bmQoKC4uY291bnQuLikvc3VtKC4uY291bnQuLiksIDIpLCB5ID0gLi5wcm9wLi4gKyAwLjAyKSkrDQogIHRoZW1lKGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KGFuZ2xlID0gOTApKSArIA0KICBzY2FsZV95X2NvbnRpbnVvdXMobGFiZWxzPXBlcmNlbnQpKyB5bGFiKCJwcm9wIikrDQogIGdndGl0bGUoImNsYXNzX2NvdW50cnkiKQ0KDQpnZ3Bsb3QoYWR1bHQyLCBhZXMoeCA9IGNsYXNzX2NvdW50cnksIGZpbGwgPSBhcy5mYWN0b3IocmVzcG9zdGEpICkpICsgDQogIGdlb21fYmFyKHBvc2l0aW9uPSJmaWxsIikgKyANCiAgdGhlbWUoYXhpcy50ZXh0LnggPSBlbGVtZW50X3RleHQoYW5nbGUgPSA5MCkpICsgDQogIHNjYWxlX3lfY29udGludW91cyhsYWJlbHM9cGVyY2VudCkrDQogIGdndGl0bGUoImNsYXNzX2NvdW50cnkiKQ0KICAgIA0KICAgIA0KYGBgDQoNCjxicj5BIGRpc3RyaWJ1acOnw6NvIGZpY291IGNvbSA1JSBwYXJhIHBhw61zZXMgYWNpbWEgZGEgbcOpZGlhIGUgNSUgcGFyYSBwYcOtc2VzIGFiYWl4byBkYSBtw6lkaWEuPC9icj4NCg0KIyBNb2RlbGFnZW0NCg0KQ29tIG5vc3NhIGEgYW7DoWxpc2UgZXhwbG9yYXTDs3JpYSBjb25jbHXDrWRhLCB2YW1vcyBkYXIgaW7DrWNpbyBhcyBlc3RhcGFzIGRhIG1vZGVsYWdlbSB1dGlsaXphbmRvIG8gZnJhbWV3b3JrIGRvIGB0aWR5bW9kZWxzYC4NCg0KIyMgQW1vc3RyYWdlbQ0KDQpGYXplbmRvIGEgc2VwYXJhw6fDo28gZG9zIGRhZG9zIGVtIHRyZWlubyBlIHRlc3RlIHBhcmEgYSBtb2RlbGFnZW0uDQoNCmBgYHtyLCBlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9DQpzZXQuc2VlZCgzMikNCg0KYWR1bHRfc3BsaXQgPC0gaW5pdGlhbF9zcGxpdChhZHVsdCwgcHJvcCA9IDAuOCwgc3RyYXRhID0gcmVzcG9zdGEpDQoNCmFkdWx0X3RyYWluIDwtIHRyYWluaW5nKGFkdWx0X3NwbGl0KQ0KYWR1bHRfdGVzdCA8LSB0ZXN0aW5nKGFkdWx0X3NwbGl0KQ0KDQpgYGANCg0KDQojIyBEYXRhIFByZXANCg0KT3MgdHJhdGFtZW50b3MgbmVjZXNzw6FyaW9zIG9ic2VydmFkb3MgbmEgQUVELCBxdWUgZm9pIGZlaXRhIHV0aWxpemFuZG8gbyBwYWNvdGUgYERhdGFFeHBsb3JlcmAgZSBhIGZ1bsOnw6NvIFtgQUVEX2JpdmBdKGh0dHBzOi8vZ2l0aHViLmNvbS9yaWNhcmRvbWF0dG9zMDUvZnVuY3Rpb25zL2Jsb2IvbWFzdGVyL2Z1bmN0aW9uX0FFRF9iaXZhcmlhZGEuUikgcXVlIGdlcmVpIHBhcmEgZW50ZW5kZXIgbyBjb21wb3J0YW1lbnRvIGRhcyB2YXJpw6F2ZWlzIGNvbSByZWxhw6fDo28gYSB2YXJpw6F2ZWwgcmVzcG9zdGEsIHNlcsOjbyBhcm1hemVuYWRvcyB1dGlsaXphbmRvIG8gcmVjaXBlcyBwYXJhIHNlciB1dGlsaXphZG8gdGFudG8gcGFyYSB0cmVpbmFyIG9zIG1vZGVsb3MgY29tbyBwYXJhIHRlc3RhciBwb3N0ZXJpb3JtZW50ZS4NCg0KDQpgYGB7cn0NCg0KYWR1bHRfcmVjaXBlIDwtIA0KICByZWNpcGUocmVzcG9zdGEgfiAuLCBkYXRhID0gYWR1bHRfdHJhaW4pICU+JSANCiAgc3RlcF9tdXRhdGUoDQogICAgDQogICAgb2NjdXBhdGlvbiA9IGNhc2Vfd2hlbigNCiAgICAgIGlzLm5hKG9jY3VwYXRpb24pIH4gIkZhcm1pbmctZmlzaGluZyIsDQogICAgICBUUlVFIH4gYXMuY2hhcmFjdGVyKG9jY3VwYXRpb24pKSwNCiAgICANCiAgICB3b3JrY2xhc3MgPSBjYXNlX3doZW4oDQogICAgICBpcy5uYSh3b3JrY2xhc3MpIH4gIk5vdC1pZGVudGlmeSIsDQogICAgICBUUlVFIH4gYXMuY2hhcmFjdGVyKHdvcmtjbGFzcykpLA0KICAgIA0KICAgIGNsYXNzX2NvdW50cnkgPSBjYXNlX3doZW4obmF0aXZlX2NvdW50cnkgJWluJSBjKCJDYW1ib2RpYSIsICJDYW5hZGEiLCAiQ2hpbmEiLCAiQ3ViYSIsICJFbmdsYW5kIiwgIkZyYW5jZSIsICJHZXJtYW55IiwgIkdyZWVjZSIsICJIb25nIiwgIkluZGlhIiwgIklyYW4iLCAiSXRhbHkiLCAiSmFwYW4iLCAiUGhpbGlwcGluZXMiLCAiU2NvdGxhbmQiLCAiVGFpd2FuIiwgIll1Z29zbGF2aWEiLCBOQSkgfiAiZ3JlYXRlcl9tZWFuIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbmF0aXZlX2NvdW50cnkgPT0gIlVuaXRlZC1TdGF0ZXMiIH4gIlVuaXRlZC1TdGF0ZXMiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICBUUlVFIH4gInNtYWxsZXJfbWVhbiIpDQogICAgLA0KICAgIA0KICAgIGNhcGl0YWxfdG90YWwgPSBjYXBpdGFsX2dhaW4gKyBjYXBpdGFsX2xvc3MNCiAgICAsIA0KICAgIA0KICAgIG1hcml0YWxfc3RhdHVzID0gY2FzZV93aGVuKA0KICAgICAgbWFyaXRhbF9zdGF0dXMgJWluJSBjKCJNYXJyaWVkLUFGLXNwb3VzZSIgLCAiTWFycmllZC1jaXYtc3BvdXNlIikgfiAiTWFycmllZCIsDQogICAgICBUUlVFIH4gYXMuY2hhcmFjdGVyKG1hcml0YWxfc3RhdHVzKSkNCiAgICAsDQogICAgDQogICAgZWR1Y2F0aW9uID0gY2FzZV93aGVuKGVkdWNhdGlvbiAlaW4lIGMoIjFzdC00dGgiLCAiNXRoLTZ0aCIsICI3dGgtOHRoIiwgIjl0aCIsICIxMHRoIiwgIjExdGgiLCAiMTJ0aCIpIH4gIkhTLW5vdC1ncmFkIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgVFJVRSB+IGFzLmNoYXJhY3RlcihlZHVjYXRpb24pKQ0KICAgICwNCiAgICANCiAgICByZWxhdGlvbnNoaXAgPSBjYXNlX3doZW4oICByZWxhdGlvbnNoaXAgJWluJSBjKCJIdXNiYW5kIiwiV2lmZSIpIH4gIk1hcnJpZWQiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFRSVUUgfiBhcy5jaGFyYWN0ZXIocmVsYXRpb25zaGlwKSkNCiAgICANCiAgKSAlPiUgDQogIHN0ZXBfcm0oaWQsIGNhcGl0YWxfZ2FpbiwgY2FwaXRhbF9sb3NzLCBuYXRpdmVfY291bnRyeSklPiUgDQogIHN0ZXBfc3RyaW5nMmZhY3RvcihhbGxfbm9taW5hbCgpKSAlPiUNCiAgc3RlcF9ub3JtYWxpemUoYWxsX251bWVyaWMoKSkgJT4lIA0KICBzdGVwX3p2KGFsbF9wcmVkaWN0b3JzKCkpICU+JQ0KICBzdGVwX25vdmVsKGFsbF9ub21pbmFsKCksIC1hbGxfb3V0Y29tZXMoKSkgJT4lIA0KICBzdGVwX2R1bW15KGFsbF9ub21pbmFsKCksIC1hbGxfb3V0Y29tZXMoKSkNCg0KICMgYmFrZShwcmVwKGFkdWx0X3JlY2lwZSksIGFkdWx0X3RyYWluKQ0KDQoNCmFkdWx0X3dmIDwtIA0KICB3b3JrZmxvdygpICU+JSANCiAgYWRkX3JlY2lwZShhZHVsdF9yZWNpcGUpDQpgYGANCg0KDQoNCiMjIENyb3NzLVZhbGlkYXRpb24NCg0KRXNwZWNpZmljYW5kbyBhIHZhbGlkYcOnw6NvIGNydXphZGE6DQoNCmBgYHtyfQ0Kc2V0LnNlZWQoMzIpDQphZHVsdF92Zm9sZCA8LSB2Zm9sZF9jdihhZHVsdF90cmFpbiwgdiA9IDUsIHN0cmF0YSA9IHJlc3Bvc3RhKQ0KYWR1bHRfdmZvbGQNCmBgYA0KDQojIyBNb2RlbG9zIHsudGFic2V0fQ0KDQpPcyBtb2RlbG9zIHF1ZSBzZXLDo28gYWp1c3RhZG9zOg0KDQogICogRGVjaXNpb24gdHJlZQ0KICAqIFJhbmRvbSBGb3Jlc3QNCiAgKiBYZ2Jvb3N0DQogIA0KIyMjIERlY2lzaW9uIHRyZWUNCg0KRXNwZWNpZmljYW5kbyBtb2RlbG86DQoNCmBgYHtyfQ0KYWR1bHRfdHJlZSA8LSANCiAgZGVjaXNpb25fdHJlZSgNCiAgICBtaW5fbiA9IHR1bmUoKSwNCiAgICBjb3N0X2NvbXBsZXhpdHkgPSB0dW5lKCksIA0KICAgIHRyZWVfZGVwdGggPSB0dW5lKCkpICU+JQ0KICBzZXRfbW9kZSgiY2xhc3NpZmljYXRpb24iKSAlPiUNCiAgc2V0X2VuZ2luZSgicnBhcnQiKQ0KDQphZHVsdF90cmVlDQpgYGANCldvcmtmbG93IHBhcmEgZGVjaXNpb24gdHJlZToNCg0KYGBge3J9DQoNCndvcmtmbG93X2FkdWx0X3RyZWUgPC0gDQogIGFkdWx0X3dmICU+JSANCiAgYWRkX21vZGVsKGFkdWx0X3RyZWUpDQoNCg0KYGBgDQoNClBhcsOibWVudHJvczoNCg0KYGBge3J9DQpoaXBlcnBhcmFtcyA8LSBwYXJhbWV0ZXJzKA0KIGFkdWx0X3RyZWUNCikNCmhpcGVycGFyYW1zDQpgYGANCg0KR3JpZDoNCg0KYGBge3J9DQpzZXQuc2VlZCgzMikNCnRyZWVfZ3JpZCA8LSBncmlkX21heF9lbnRyb3B5KGhpcGVycGFyYW1zLCBzaXplID0gMTApDQoNCmBgYA0KDQoNCkVmZXR1YW5kbyB0dW5hZ2VtIGRlIGhpcGVycGFyw6JtZXRyb3M6DQoNCmBgYHtyLCBlY2hvPSBUUlVFfQ0KDQp0cmVlX3R1bmUgPC0gDQogIHdvcmtmbG93X2FkdWx0X3RyZWUgJT4lIA0KICB0dW5lX2dyaWQoDQogICAgcmVzYW1wbGVzID0gYWR1bHRfdmZvbGQsDQogICAgZ3JpZCA9IHRyZWVfZ3JpZCwNCiAgICBjb250cm9sID0gY29udHJvbF9ncmlkKHNhdmVfcHJlZCA9IFRSVUUsIHZlcmJvc2UgPSBULCBhbGxvd19wYXIgPSBGKSwNCiAgICBtZXRyaWNzID0gbWV0cmljX3NldChyb2NfYXVjKQ0KICApDQoNCmBgYA0KDQpgYGB7cn0NCmF1dG9wbG90KHRyZWVfdHVuZSkNCnNob3dfYmVzdCh0cmVlX3R1bmUsICJyb2NfYXVjIikNCg0KdHJlZV9iZXN0X2hpcGVycGFyYW1zIDwtIHNlbGVjdF9iZXN0KHRyZWVfdHVuZSkgIzEuMTc1NzY0ZS0wNQk4CTEzCU1vZGVsMDcgKHJvY19hdWMgPSAwLjkwMDg0NjYpDQp0cmVlX2Jlc3RfaGlwZXJwYXJhbXMNCg0KYGBgDQoNCkZpbmFsaXphbmRvIFdGOg0KDQpgYGB7cn0NCndvcmtmbG93X3RyZWVfZmluYWwgPC0gZmluYWxpemVfd29ya2Zsb3coDQogIHdvcmtmbG93X2FkdWx0X3RyZWUsDQogIHRyZWVfYmVzdF9oaXBlcnBhcmFtcw0KKQ0KDQp3b3JrZmxvd190cmVlX2ZpbmFsDQpgYGANCg0KDQpWZXJpZmljYW5kbyBpbXBvcnTDom5jaWEgZG9zIGF0cmlidXRvczoNCg0KYGBge3J9DQp3b3JrZmxvd190cmVlX2ZpbmFsICU+JQ0KICBmaXQoYWR1bHRfdHJhaW4pICU+JQ0KICBwdWxsX3dvcmtmbG93X2ZpdCgpICU+JQ0KICB2aXA6OnZpcChnZW9tID0gImNvbCIpDQpgYGANCg0KDQpNb2RlbG8gZmluYWw6DQoNCmBgYHtyfQ0KDQp0cmVlX2ZpbmFsIDwtIGxhc3RfZml0KHdvcmtmbG93X3RyZWVfZmluYWwsIGFkdWx0X3NwbGl0KQ0KY29sbGVjdF9tZXRyaWNzKHRyZWVfZmluYWwpICMgcm9jX2F1YyA9IDAuODk0NzYyNA0KDQpgYGANCg0KDQoNCiMjIyBSYW5kb20gRm9yZXN0DQoNCkVzcGVjaWZpY2FuZG8gbW9kZWxvOg0KDQpgYGB7cn0NCmFkdWx0X3JmIDwtIA0KICByYW5kX2ZvcmVzdCgNCiAgICBtaW5fbiA9IHR1bmUoKSwNCiAgICBtdHJ5ID0gdHVuZSgpLA0KICAgIHRyZWVzID0gdHVuZSgpKSAlPiUNCiAgc2V0X21vZGUoImNsYXNzaWZpY2F0aW9uIikgJT4lDQogIHNldF9lbmdpbmUoInJhbmRvbUZvcmVzdCIpDQoNCmFkdWx0X3JmDQpgYGANCg0KV29ya2Zsb3cgcGFyYSByYW5kb20gZm9yZXN0Og0KDQpgYGB7cn0NCg0Kd29ya2Zsb3dfYWR1bHRfcmYgPC0gDQogIGFkdWx0X3dmICU+JSANCiAgYWRkX21vZGVsKGFkdWx0X3JmKQ0KDQoNCmBgYA0KDQoNCkdyaWQ6DQoNCmBgYHtyfQ0Kc2V0LnNlZWQoMzIpDQoNCnJmX2dyaWQgPC0gcGFyYW1ldGVycyhhZHVsdF9yZikgJT4lIA0KICBmaW5hbGl6ZShiYWtlKHByZXAoYWR1bHRfcmVjaXBlKSwgYWR1bHRfdHJhaW4pKSAlPiUgDQogIGdyaWRfbWF4X2VudHJvcHkoc2l6ZSA9IDEwKQ0KDQoNCmBgYA0KDQpFZmV0dWFuZG8gdHVuYWdlbSBkZSBoaXBlcnBhcsOibWV0cm9zOg0KDQpgYGB7cn0NCg0KbGlicmFyeShkb1BhcmFsbGVsKQ0KbGlicmFyeSgiZG9GdXR1cmUiKQ0KDQphbGxfY29yZXMgPC0gcGFyYWxsZWw6OmRldGVjdENvcmVzKGxvZ2ljYWwgPSBGQUxTRSkgLSAxDQpyZWdpc3RlckRvRnV0dXJlKCkNCmNsIDwtIG1ha2VDbHVzdGVyKGFsbF9jb3JlcykNCnBsYW4oZnV0dXJlOjpjbHVzdGVyLCB3b3JrZXJzID0gY2wpDQpnZXREb1BhcldvcmtlcnMoKQ0KDQpzZXQuc2VlZCgxMjMpDQpyZl90dW5lPC0gDQogIHdvcmtmbG93X2FkdWx0X3JmICU+JSANCiAgdHVuZV9ncmlkKA0KICAgIHJlc2FtcGxlcyA9IGFkdWx0X3Zmb2xkLA0KICAgIGdyaWQgPSByZl9ncmlkLA0KICAgIGNvbnRyb2wgPSBjb250cm9sX2dyaWQoc2F2ZV9wcmVkID0gVFJVRSwgdmVyYm9zZSA9IFQsIGFsbG93X3BhciA9IFQpLA0KICAgIG1ldHJpY3MgPSBtZXRyaWNfc2V0KHJvY19hdWMpDQogICkNCg0KZm9yZWFjaDo6cmVnaXN0ZXJEb1NFUSgpDQpgYGANCg0KYGBge3J9DQphdXRvcGxvdChyZl90dW5lKQ0Kc2hvd19iZXN0KHJmX3R1bmUsInJvY19hdWMiKQ0KDQpyZl9iZXN0X2hpcGVycGFyYW1zIDwtIHNlbGVjdF9iZXN0KHJmX3R1bmUpIA0KcmZfYmVzdF9oaXBlcnBhcmFtcyAjbXRyeSA9IDQyCXRyZWVzID0gMTQwNgkgbWluX24gPSAyMSAJTW9kZWwxMCAocm9jX2F1YyA9IDAuOTExNTE0MykNCg0KYGBgDQoNCkZpbmFsaXphbmRvIFdGOg0KDQpgYGB7cn0NCndvcmtmbG93X3JmX2ZpbmFsIDwtIGZpbmFsaXplX3dvcmtmbG93KA0KICB3b3JrZmxvd19hZHVsdF9yZiwNCiAgcmZfYmVzdF9oaXBlcnBhcmFtcw0KKQ0KDQp3b3JrZmxvd19yZl9maW5hbA0KYGBgDQoNClZlcmlmaWNhbmRvIGltcG9ydMOibmNpYSBkb3MgYXRyaWJ1dG9zOg0KDQpgYGB7cn0NCndvcmtmbG93X3JmX2ZpbmFsICU+JQ0KICBmaXQoYWR1bHRfdHJhaW4pICU+JQ0KICBwdWxsX3dvcmtmbG93X2ZpdCgpICU+JQ0KICB2aXA6OnZpcChnZW9tID0gImNvbCIpDQoNCmBgYA0KDQoNCk1vZGVsbyBmaW5hbDoNCg0KYGBge3J9DQoNCnJmX2ZpbmFsIDwtIGxhc3RfZml0KHdvcmtmbG93X3JmX2ZpbmFsLCBhZHVsdF9zcGxpdCkNCmNvbGxlY3RfbWV0cmljcyhyZl9maW5hbCkgI3JvY19hdWMgPSAwLjkwNzI4ODUNCg0KYGBgDQoNCiMjIyBYZ2Jvb3N0DQoNClR1bmFuZG8gbXRyeSwgdHJlZXMgZSBzYW1wbGUgc2l6ZToNCg0KYGBge3J9DQphZHVsdF94Z2IgPC0gDQogIGJvb3N0X3RyZWUoDQogICBtdHJ5ID0gdHVuZSgpLCANCiAgdHJlZXMgPSB0dW5lKCksIA0KICBtaW5fbiA9IHR1bmUoKSwgDQogIHRyZWVfZGVwdGggPSB0dW5lKCksDQogICMgbG9zc19yZWR1Y3Rpb24gPSB0dW5lKCksIA0KICBsZWFybl9yYXRlID0gdHVuZSgpLCANCiAgIyBzYW1wbGVfc2l6ZSA9IHR1bmUoKQ0KICApICU+JQ0KICBzZXRfbW9kZSgiY2xhc3NpZmljYXRpb24iKSAlPiUNCiAgc2V0X2VuZ2luZSgieGdib29zdCIpDQoNCmFkdWx0X3hnYg0KYGBgDQoNCldvcmtmbG93IHBhcmEgWGdib29zdDoNCg0KYGBge3J9DQoNCndvcmtmbG93X2FkdWx0X3hnYiA8LSANCiAgYWR1bHRfd2YgJT4lIA0KICBhZGRfbW9kZWwoYWR1bHRfeGdiKQ0KYGBgDQoNCkdyaWQ6DQoNCmBgYHtyfQ0Kc2V0LnNlZWQoMzIpDQoNCnhnYl9ncmlkIDwtIHBhcmFtZXRlcnMoYWR1bHRfeGdiKSAlPiUNCiAgICBmaW5hbGl6ZShiYWtlKHByZXAoYWR1bHRfcmVjaXBlKSxhZHVsdF90cmFpbikpICU+JQ0KICAgIGdyaWRfbWF4X2VudHJvcHkoc2l6ZSA9IDIwKQ0KDQp4Z2JfZ3JpZA0KDQoNCmBgYA0KRWZldHVhbmRvIHR1bmFnZW0gZGUgaGlwZXJwYXLDom1ldHJvczoNCg0KYGBge3J9DQoNCmFsbF9jb3JlcyA8LSBwYXJhbGxlbDo6ZGV0ZWN0Q29yZXMobG9naWNhbCA9IEZBTFNFKSAtIDENCg0KcmVnaXN0ZXJEb0Z1dHVyZSgpDQpjbCA8LSBtYWtlQ2x1c3RlcihhbGxfY29yZXMpDQpwbGFuKGZ1dHVyZTo6Y2x1c3Rlciwgd29ya2VycyA9IGNsKQ0KZ2V0RG9QYXJXb3JrZXJzKCkNCg0KIyBncmlkIHNlYXJjaA0KaW5pIDwtIFN5cy50aW1lKCkNCnhnYl90dW5lIDwtDQogIHdvcmtmbG93X2FkdWx0X3hnYiAlPiUNCiAgICB0dW5lX2dyaWQoDQogICAgICAgIHJlc2FtcGxlcyA9IGFkdWx0X3Zmb2xkLA0KICAgICAgICBncmlkID0geGdiX2dyaWQsDQogICAgICAgIGNvbnRyb2wgPSBjb250cm9sX2dyaWQodmVyYm9zZSA9IFRSVUUpLA0KICAgICAgICBtZXRyaWNzID0gbWV0cmljX3NldChyb2NfYXVjKQ0KICAgICkNClN5cy50aW1lKCktIGluaSAjVGltZSBkaWZmZXJlbmNlIG9mIDM5Ljk4NDQgbWlucyhwYXJhbGxlbCkNCg0KZm9yZWFjaDo6cmVnaXN0ZXJEb1NFUSgpDQpgYGANCg0KDQpgYGB7cn0NCmF1dG9wbG90KHhnYl90dW5lKQ0Kc2hvd19iZXN0KHhnYl90dW5lLCJyb2NfYXVjIikNCg0KeGdiX2Jlc3RfaGlwZXJwYXJhbXMgPC0gc2VsZWN0X2Jlc3QoeGdiX3R1bmUpDQp4Z2JfYmVzdF9oaXBlcnBhcmFtcyAjMzQJMTMwOQk1CTEwCTAuMDEwNjQ0NTA1CU1vZGVsMTMgKHJvY19hdWMgPSAwLjkyNzA1MzIpDQoNCmBgYA0KRmluYWxpemFuZG8gV0Y6DQoNCmBgYHtyfQ0Kd29ya2Zsb3dfeGdiX2ZpbmFsIDwtIGZpbmFsaXplX3dvcmtmbG93KA0KICB3b3JrZmxvd19hZHVsdF94Z2IsDQogIHhnYl9iZXN0X2hpcGVycGFyYW1zIA0KKQ0KDQp3b3JrZmxvd194Z2JfZmluYWwNCmBgYA0KDQpWZXJpZmljYW5kbyBpbXBvcnTDom5jaWEgZG9zIGF0cmlidXRvczoNCg0KYGBge3J9DQp3b3JrZmxvd194Z2JfZmluYWwgJT4lDQogIGZpdChhZHVsdF90cmFpbikgJT4lDQogIHB1bGxfd29ya2Zsb3dfZml0KCkgJT4lDQogIHZpcDo6dmlwKGdlb20gPSAiY29sIikNCmBgYA0KDQpNb2RlbG8gZmluYWw6DQoNCmBgYHtyfQ0KDQp4Z2JfZmluYWwgPC0gbGFzdF9maXQod29ya2Zsb3dfeGdiX2ZpbmFsLCBhZHVsdF9zcGxpdCkNCmNvbGxlY3RfbWV0cmljcyh4Z2JfZmluYWwpDQoNCmBgYA0KDQojIyMgWGdib29zdDINCg0KVHVuYW5kbyBtdHJ5LCB0cmVlcyBlIHNhbXBsZSBzaXplOg0KDQpgYGB7cn0NCmFkdWx0X3hnYjIgPC0gDQogIGJvb3N0X3RyZWUoDQogICBtdHJ5ID0gMzQsIA0KICB0cmVlcyA9IDEzMDksIA0KICBtaW5fbiA9IDUsIA0KICB0cmVlX2RlcHRoID0gMTAsDQogIGxvc3NfcmVkdWN0aW9uID0gdHVuZSgpLA0KICBsZWFybl9yYXRlID0gMC4wMTA2NDQ1LCANCiAgc2FtcGxlX3NpemUgPSB0dW5lKCkNCiAgKSAlPiUNCiAgc2V0X21vZGUoImNsYXNzaWZpY2F0aW9uIikgJT4lDQogIHNldF9lbmdpbmUoInhnYm9vc3QiKQ0KDQphZHVsdF94Z2IyDQoNCmBgYA0KDQpXb3JrZmxvdyBwYXJhIFhnYm9vc3Q6DQoNCmBgYHtyfQ0KDQp3b3JrZmxvd19hZHVsdF94Z2IyIDwtIA0KICBhZHVsdF93ZiAlPiUgDQogIGFkZF9tb2RlbChhZHVsdF94Z2IyKQ0KYGBgDQoNCkdyaWQ6DQoNCmBgYHtyfQ0Kc2V0LnNlZWQoMzIpDQoNCnhnYl9ncmlkMiA8LSBwYXJhbWV0ZXJzKGFkdWx0X3hnYjIpICU+JQ0KICAgIGdyaWRfbWF4X2VudHJvcHkoc2l6ZSA9IDIwKQ0KDQp4Z2JfZ3JpZDINCg0KDQpgYGANCkVmZXR1YW5kbyB0dW5hZ2VtIGRlIGhpcGVycGFyw6JtZXRyb3M6DQoNCmBgYHtyfQ0KDQphbGxfY29yZXMgPC0gcGFyYWxsZWw6OmRldGVjdENvcmVzKGxvZ2ljYWwgPSBGQUxTRSkgLSAxDQoNCnJlZ2lzdGVyRG9GdXR1cmUoKQ0KY2wgPC0gbWFrZUNsdXN0ZXIoYWxsX2NvcmVzKQ0KcGxhbihmdXR1cmU6OmNsdXN0ZXIsIHdvcmtlcnMgPSBjbCkNCmdldERvUGFyV29ya2VycygpDQoNCiMgZ3JpZCBzZWFyY2gNCmluaSA8LSBTeXMudGltZSgpDQp4Z2JfdHVuZTIgPC0NCiAgd29ya2Zsb3dfYWR1bHRfeGdiMiAlPiUNCiAgICB0dW5lX2dyaWQoDQogICAgICAgIHJlc2FtcGxlcyA9IGFkdWx0X3Zmb2xkLA0KICAgICAgICBncmlkID0geGdiX2dyaWQyLA0KICAgICAgICBjb250cm9sID0gY29udHJvbF9ncmlkKHZlcmJvc2UgPSBUUlVFKSwNCiAgICAgICAgbWV0cmljcyA9IG1ldHJpY19zZXQocm9jX2F1YykNCiAgICApDQpTeXMudGltZSgpLSBpbmkgI1RpbWUgZGlmZmVyZW5jZSBvZiAzOS45ODQ0IG1pbnMocGFyYWxsZWwpDQoNCmZvcmVhY2g6OnJlZ2lzdGVyRG9TRVEoKQ0KYGBgDQoNCg0KYGBge3J9DQphdXRvcGxvdCh4Z2JfdHVuZTIpDQpzaG93X2Jlc3QoeGdiX3R1bmUyLCJyb2NfYXVjIikNCg0KeGdiMl9iZXN0X2hpcGVycGFyYW1zIDwtIHNlbGVjdF9iZXN0KHhnYl90dW5lMikNCnhnYjJfYmVzdF9oaXBlcnBhcmFtcyAjMi4wODc2NDdlKzAwCTAuODY5Njk2NwkgKHJvY19hdWMgPSAwLjkyNjg5NDcpDQoNCmBgYA0KRmluYWxpemFuZG8gV0Y6DQoNCmBgYHtyfQ0Kd29ya2Zsb3dfeGdiX2ZpbmFsMiA8LSBmaW5hbGl6ZV93b3JrZmxvdygNCiAgd29ya2Zsb3dfYWR1bHRfeGdiMiwNCiAgeGdiMl9iZXN0X2hpcGVycGFyYW1zIA0KKQ0KDQp3b3JrZmxvd194Z2JfZmluYWwyDQpgYGANCg0KVmVyaWZpY2FuZG8gaW1wb3J0w6JuY2lhIGRvcyBhdHJpYnV0b3M6DQoNCmBgYHtyfQ0Kd29ya2Zsb3dfeGdiX2ZpbmFsMiAlPiUNCiAgZml0KGFkdWx0X3RyYWluKSAlPiUNCiAgcHVsbF93b3JrZmxvd19maXQoKSAlPiUNCiAgdmlwOjp2aXAoZ2VvbSA9ICJjb2wiKQ0KYGBgDQoNCk1vZGVsbyBmaW5hbDoNCg0KYGBge3J9DQoNCnhnYjJfZmluYWwgPC0gbGFzdF9maXQod29ya2Zsb3dfeGdiX2ZpbmFsMiwgYWR1bHRfc3BsaXQpDQpjb2xsZWN0X21ldHJpY3MoeGdiMl9maW5hbCkNCg0KYGBgDQoNCiMjIyBYZ2Jvb3N0Ng0KDQpUdW5hbmRvIG10cnksIHRyZWVzIGUgc2FtcGxlIHNpemU6DQoNCmBgYHtyfQ0KYWR1bHRfeGdiNiA8LSANCiAgYm9vc3RfdHJlZSgNCiAgICB0cmVlX2RlcHRoID0gNywgDQogICAgdHJlZXMgPSAxMzQ3LA0KICAgIGxlYXJuX3JhdGUgPSAxLjU2NjY5M2UtMDIsDQogICAgbXRyeSA9IDMzLA0KICAgIG1pbl9uID0gNQ0KICApICU+JQ0KICBzZXRfbW9kZSgiY2xhc3NpZmljYXRpb24iKSAlPiUNCiAgc2V0X2VuZ2luZSgieGdib29zdCIpDQoNCmFkdWx0X3hnYjYNCg0KYGBgDQoNCg0KV29ya2Zsb3cgcGFyYSBYZ2Jvb3N0Og0KDQpgYGB7cn0NCg0Kd29ya2Zsb3dfYWR1bHRfeGdiNiA8LSANCiAgYWR1bHRfd2YgJT4lIA0KICBhZGRfbW9kZWwoYWR1bHRfeGdiNikNCg0KDQpgYGANCg0KRmluYWxpemFuZG8gV0Y6DQoNCmBgYHtyfQ0Kd29ya2Zsb3dfeGdiX2ZpbmFsNiA8LSBmaW5hbGl6ZV93b3JrZmxvdygNCiAgd29ya2Zsb3dfYWR1bHRfeGdiNiwNCiAgcGFyYW1ldGVycyhhZHVsdF94Z2I2KSANCikNCndvcmtmbG93X3hnYl9maW5hbDYNCg0KYGBgDQoNCg0KVmVyaWZpY2FuZG8gaW1wb3J0w6JuY2lhIGRvcyBhdHJpYnV0b3M6DQoNCmBgYHtyfQ0Kd29ya2Zsb3dfeGdiX2ZpbmFsNiAlPiUNCiAgZml0KGFkdWx0X3RyYWluKSAlPiUNCiAgcHVsbF93b3JrZmxvd19maXQoKSAlPiUNCiAgdmlwOjp2aXAoZ2VvbSA9ICJjb2wiKQ0KYGBgDQoNCk1vZGVsbyBmaW5hbDoNCg0KYGBge3J9DQoNCnhnYl9maW5hbDYgPC0gbGFzdF9maXQod29ya2Zsb3dfeGdiX2ZpbmFsNiwgYWR1bHRfc3BsaXQpDQpjb2xsZWN0X21ldHJpY3MoeGdiX2ZpbmFsNikNCg0KDQoNCmBgYA0KDQojIFNjb3JhZ2VtIHBhcmEgc3VibWV0ZXIgcmVzdWx0YWRvDQoNCmBgYHtyfQ0KDQphZHVsdF92YWwgPC0gcmVhZHI6OnJlYWRfcmRzKCJhZHVsdF92YWwucmRzIikNCg0KeGdib29zdF9tb2RlbG9fZmluYWwgPC0gYWR1bHRfeGdiICU+JSANCiAgICBmaW5hbGl6ZV9tb2RlbCh4Z2JfYmVzdF9oaXBlcnBhcmFtcykNCg0KYWR1bHRfZml0IDwtIA0KICBmaXQoeGdib29zdF9tb2RlbG9fZmluYWwsDQogICAgZm9ybXVsYSA9IHJlc3Bvc3RhIH4gLiwgIA0KICAgIGRhdGEgPSBiYWtlKHByZXAoYWR1bHRfcmVjaXBlKSwgbmV3X2RhdGEgPSBhZHVsdCkpDQoNCmFkdWx0X3ZhbCRtb3JlX3RoYW5fNTBrIDwtIA0KICBwcmVkaWN0KGFkdWx0X2ZpdCwgDQogICAgICAgICAgYmFrZShwcmVwKGFkdWx0X3JlY2lwZSksIG5ld19kYXRhID0gYWR1bHRfdmFsKSwNCiAgICAgICAgICB0eXBlID0gInByb2IiKSRgLnByZWRfPjUwS2ANCmBgYA0KDQpNYXRyaXogZGUgY29uZnVzw6NvOg0KDQpgYGB7cn0NCmxpYnJhcnkoImUxMDcxIikNCg0KYWR1bHRfdmFsICU+JSANCiAgdHJhbnNtdXRlKHJlc3Bvc3RhID0gZmFjdG9yKHJlc3Bvc3RhLCBsZXZlbHMgPSBjKCI+NTBLIiwgIjw9NTBLIikpLCANCiAgICAgICAgICAgIG1vcmVfdGhhbl81MGsgPSBpZmVsc2UobW9yZV90aGFuXzUwayA+IDAuNSwgIj41MEsiLCAiPD01MEsiKSAlPiUgDQogICAgICAgICAgICAgIGZhY3RvcihsZXZlbHMgPSBjKCI+NTBLIiwgIjw9NTBLIikpKSAlPiUgDQogIHRhYmxlKCkgJT4lIA0KICBjYXJldDo6Y29uZnVzaW9uTWF0cml4KCkNCg0KbGlicmFyeSgncFJPQycpDQpwbG90KHJvYyhhZHVsdF92YWwkcmVzcG9zdGEsIGFkdWx0X3ZhbCRtb3JlX3RoYW5fNTBrKSkNCg0KDQphZHVsdF92YWwgJT4lIA0KICByb2NfYXVjKHRydXRoID0gYXMuZmFjdG9yKHJlc3Bvc3RhKSwgbW9yZV90aGFuXzUwaykNCg0KI3hnYiB3aXRoIG5ldyBzdGVwX211dGF0ZTogMC45MjU1ODINCiN4Z2IyIHdpdGggbmV3IHN0ZXBfbXV0YXRlOiAwLjkyNjI4NjMNCiN4Z2I1OiAwLjkyNDc2NTkNCiN4Z2I1XzIgdHVubmFuZG8gYXBlbmFzIDUgcGFyYW1zOiAwLjkyNzc3NTENCiN4Z2I2OiAgMC45Mjg4MjI5DQoNCmBgYA0KDQoNCg0KDQoNCg0KDQo=